<html>
<head><meta charset="utf-8"><title>OsStr and WTF-8 · t-libs · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/index.html">t-libs</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html">OsStr and WTF-8</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="242814615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242814615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jacob Pratt <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242814615">(Jun 15 2021 at 23:57)</a>:</h4>
<p>Has it ever been considered to make WTF-8 a stable format as the backing data type? This would allow things like <code>.as_bytes()</code> for all OS's, not just Unix via the extension trait.</p>



<a name="242819808"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242819808" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242819808">(Jun 16 2021 at 01:20)</a>:</h4>
<p><span class="user-mention" data-user-id="245610">@Jacob Pratt</span> It's been discussed. I'd love to see that happen.</p>



<a name="242819838"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242819838" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242819838">(Jun 16 2021 at 01:21)</a>:</h4>
<p>Though to clarify something, OsStr/OsString would never be guaranteed to have WTF-8 on all platforms.</p>



<a name="242819843"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242819843" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242819843">(Jun 16 2021 at 01:22)</a>:</h4>
<p>But we could guarantee bytes on all platforms.</p>



<a name="242819934"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242819934" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jacob Pratt <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242819934">(Jun 16 2021 at 01:23)</a>:</h4>
<p>Yeah, of course. Just guarantee <em>something</em> that is either UTF-8 or WTF-8 (I think that's exhaustive?). Would an RFC be needed for this? I know WTF-8 has a spec.</p>



<a name="242820861"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242820861" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> BurntSushi <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242820861">(Jun 16 2021 at 01:40)</a>:</h4>
<p>SimonSapin has thoughts about this. Is he on zulip?</p>
<p>There was an rfc about tweaking wtf8 but i don't think it is implemented yet. Iirc, we wouldn't be able to do something like that if we had exposed wtf8.</p>
<p>I selfishly very much want to expose the internal representation, but i definitely share Simon's concerns too.</p>



<a name="242820885"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242820885" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> BurntSushi <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242820885">(Jun 16 2021 at 01:41)</a>:</h4>
<p>(without exposing the internal representation, it is a real pain to run regexes or globs on paths)</p>



<a name="242821425"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242821425" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242821425">(Jun 16 2021 at 01:50)</a>:</h4>
<p><span class="user-mention silent" data-user-id="245610">Jacob Pratt</span> <a href="#narrow/stream/219381-t-libs/topic/OsStr.20and.20WTF-8/near/242819934">said</a>:</p>
<blockquote>
<p>Yeah, of course. Just guarantee <em>something</em> that is either UTF-8 or WTF-8 (I think that's exhaustive?). Would an RFC be needed for this? I know WTF-8 has a spec.</p>
</blockquote>
<p>You're not guaranteed that it's either UTF-8 or WTF-8.</p>



<a name="242821430"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242821430" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242821430">(Jun 16 2021 at 01:50)</a>:</h4>
<p>On Windows it's WTF-8. On UNIX platforms, it's bytes, no guarantee about encoding.</p>



<a name="242821445"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242821445" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242821445">(Jun 16 2021 at 01:51)</a>:</h4>
<p>You're guaranteed in both cases that you can convert UTF-8 to an OsString.</p>



<a name="242822106"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242822106" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lokathor <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242822106">(Jun 16 2021 at 02:03)</a>:</h4>
<p>so doesn't every platform have bytes then? All platforms can at least do &amp;OsStr -&gt; &amp;[u8], they just can't necessarily do &amp;mut OsStr -&gt; &amp;mut [u8]?</p>



<a name="242822120"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242822120" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lokathor <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242822120">(Jun 16 2021 at 02:03)</a>:</h4>
<p>like, how is wtf-8 different from "bytes with no encoding guarantee"?</p>



<a name="242822219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242822219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242822219">(Jun 16 2021 at 02:05)</a>:</h4>
<p><span class="user-mention" data-user-id="224471">@Lokathor</span> I agree; that's exactly the interface we can offer.</p>



<a name="242822244"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242822244" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242822244">(Jun 16 2021 at 02:06)</a>:</h4>
<p>We should allow an infallible conversion from OsStr to [u8], and a fallible conversion from [u8] to OsStr.</p>



<a name="242822291"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242822291" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242822291">(Jun 16 2021 at 02:06)</a>:</h4>
<p>(That conversion can only fail on Windows.)</p>



<a name="242825030"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242825030" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242825030">(Jun 16 2021 at 03:01)</a>:</h4>
<p>that conversion sounds easy to misuse, if it is not os-specific, since the encoding of those bytes is os-specific</p>



<a name="242825495"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242825495" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jacob Pratt <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242825495">(Jun 16 2021 at 03:09)</a>:</h4>
<blockquote>
<p>You're not guaranteed that it's either UTF-8 or WTF-8.</p>
</blockquote>
<p>What else can it be (currently)? Though I was more looking for something like <span class="user-mention" data-user-id="224471">@Lokathor</span> mentioned — just a way to get <em>some</em> bytes without caring what they are.</p>



<a name="242825839"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242825839" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242825839">(Jun 16 2021 at 03:15)</a>:</h4>
<p>Both UTF-8 and WTF-8 have invalid byte sequences, right? On linux there are no invalid byte sequences except possibly those containing <code>/</code> in weird places</p>



<a name="242826929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242826929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242826929">(Jun 16 2021 at 03:37)</a>:</h4>
<p><span class="user-mention" data-user-id="245610">@Jacob Pratt</span> On UNIX systems, OS strings can contain anything but NUL, and path components can contain anything but NUL and <code>/</code> (since <code>/</code> can only separate directories).</p>



<a name="242826987"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242826987" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242826987">(Jun 16 2021 at 03:38)</a>:</h4>
<p>So you can name a directory <code>\x99\xAA\xBB\xCC</code>, which is not valid UTF-8 or WTF-8.</p>



<a name="242827028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827028">(Jun 16 2021 at 03:39)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/219381-t-libs/topic/OsStr.20and.20WTF-8/near/242825030">said</a>:</p>
<blockquote>
<p>that conversion sounds easy to misuse, if it is not os-specific, since the encoding of those bytes is os-specific</p>
</blockquote>
<p>It's still valid to do things like "does this end in <code>.txt</code>" or "split this on <code>.</code>".</p>



<a name="242827040"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827040" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827040">(Jun 16 2021 at 03:39)</a>:</h4>
<p>Basically, it'd be reasonable to treat it as a <code>bstr::BString</code>.</p>



<a name="242827087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827087">(Jun 16 2021 at 03:40)</a>:</h4>
<p>Don't make any assumptions about arbitrary bytes, but you can operate on specific bytes you recognize.</p>



<a name="242827121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827121">(Jun 16 2021 at 03:41)</a>:</h4>
<p>are you sure that all platforms can commit to ascii as ascii? I would guess that some systems use UTF-16 or something like it (don't some windows versions do this?)</p>



<a name="242827172"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827172" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827172">(Jun 16 2021 at 03:42)</a>:</h4>
<p>that is, I think the obvious "does this end in <code>.txt</code>" algorithm is likely to be incorrect for some platforms</p>



<a name="242827485"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827485" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Chris Denton <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827485">(Jun 16 2021 at 03:50)</a>:</h4>
<p>Rust doesn't store Windows strings in their native form. Rust automatically converts Windows UTF-16 to and from UTF-8 for every OS API call. But it uses WTF-8 to account for the possibility of broken UTF-16.  So, in Rust, the Windows bytes are ASCII compatible. I'd guess that at this point <code>OsStr</code> being ASCII compatible is an explicit feature of the standard library.</p>



<a name="242827517"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827517" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827517">(Jun 16 2021 at 03:51)</a>:</h4>
<p><span class="user-mention silent" data-user-id="260325">Chris Denton</span> <a href="#narrow/stream/219381-t-libs/topic/OsStr.20and.20WTF-8/near/242827485">said</a>:</p>
<blockquote>
<p>I'd guess that at this point <code>OsStr</code> being ASCII compatible is an explicit feature of the standard library.</p>
</blockquote>
<p>Exactly. I think we should make a decision about whether to commit to that.</p>



<a name="242827518"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827518" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827518">(Jun 16 2021 at 03:51)</a>:</h4>
<p>And I think we should.</p>



<a name="242827602"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827602" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827602">(Jun 16 2021 at 03:53)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/219381-t-libs/topic/OsStr.20and.20WTF-8/near/242827172">said</a>:</p>
<blockquote>
<p>that is, I think the obvious "does this end in <code>.txt</code>" algorithm is likely to be incorrect for some platforms</p>
</blockquote>
<p>Well, to be clear, that specific example only makes sense on platforms for which file extensions make sense. But, for instance, you could ask if one <code>OsStr</code> is a suffix of another <code>OsStr</code>, which would then let you process things supplied by the user such as via environment variables or command-line arguments.</p>



<a name="242827712"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242827712" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242827712">(Jun 16 2021 at 03:55)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/219381-t-libs/topic/OsStr.20and.20WTF-8/near/242827121">said</a>:</p>
<blockquote>
<p>are you sure that all platforms can commit to ascii as ascii?</p>
</blockquote>
<p>We'd have to do some translation if Rust ever ran on an EBCDIC platform; I can live with that.</p>



<a name="242864567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242864567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242864567">(Jun 16 2021 at 11:02)</a>:</h4>
<p>shift-jis encoded names can't be blindly treated as ascii since it's a multi-byte encoding where the second byte could accidentally match ascii</p>



<a name="242885089"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242885089" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242885089">(Jun 16 2021 at 14:04)</a>:</h4>
<p>At this point, we're already allowing people to treat OsString as ASCII, by calling some of the new functions on it.</p>



<a name="242887588"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242887588" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> BurntSushi <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242887588">(Jun 16 2021 at 14:21)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="219747">@Simon Sapin</span> (i tried cc'ing you from mobile last night, but your name didn't come up, weird)</p>



<a name="242887727"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242887727" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242887727">(Jun 16 2021 at 14:21)</a>:</h4>
<p>In any case, we don't have to treat it as ASCII to expose bytes.</p>



<a name="242918589"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242918589" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242918589">(Jun 16 2021 at 18:00)</a>:</h4>
<p><span class="user-mention silent" data-user-id="260325">Chris Denton</span> <a href="#narrow/stream/219381-t-libs/topic/OsStr.20and.20WTF-8/near/242827485">said</a>:</p>
<blockquote>
<p>I'd guess that at this point <code>OsStr</code> being ASCII compatible is an explicit feature of the standard library.</p>
</blockquote>
<p>No, on Unix <code>OsStr</code> is arbitrary bytes (without nul) and there is no guarantee at all about how to meaningfully interpret those bytes as text. You could argue that "almost all" systems these days in practice use some ASCII-compatible encoding, but it’s the exact same argument as "almost all" systems use UTF-8</p>



<a name="242919752"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242919752" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242919752">(Jun 16 2021 at 18:09)</a>:</h4>
<p>That's true. I think it'd be reasonable for us to standardize that it's bytes, and also tell people "it's often ASCII, here are some functions for interpreting it as ASCII, but here are some considerations where you <em>might</em> not want to".</p>



<a name="242920099"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242920099" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242920099">(Jun 16 2021 at 18:11)</a>:</h4>
<p>Back to the original topic of exposing the WTF-8 bytes of <code>OsStr</code> on Windows, I’m conflicted. It would make a lot of things easier, but at the same time I feel that WTF-8 is an abomination that I inflicted on the world by necessity but that should never have existed. At the moment its use in <code>OsStr</code> is somewhat "contained" but with if exposed in public APIs it will eventually propagate into various file formats and network protocols, sometimes by accident, and generations of future developers will be plagued with having to deal with it in new code for backward-compatibility</p>



<a name="242920246"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242920246" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242920246">(Jun 16 2021 at 18:12)</a>:</h4>
<p>On a more pragmatic side, documenting the format as exactly WTF-8 would prevent implementing <a href="https://github.com/rust-lang/rust/issues/49802">https://github.com/rust-lang/rust/issues/49802</a> (which is an accepted but not-yet-implemented RFC)</p>



<a name="242920327"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242920327" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242920327">(Jun 16 2021 at 18:13)</a>:</h4>
<p>We can document it as "some unspecified superset of UTF-8" but that’s rather weak against users (accidentally or not) depending on WTF-8 details</p>



<a name="242922359"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242922359" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242922359">(Jun 16 2021 at 18:29)</a>:</h4>
<p>/me would be fine with not actually guaranteeing WTF-8.</p>



<a name="242922366"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242922366" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242922366">(Jun 16 2021 at 18:29)</a>:</h4>
<p>Would there really be much of a benefit of guaranteeing wtf8? for cross-platform code you'd still have to deal with bags of bytes</p>



<a name="242922371"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242922371" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242922371">(Jun 16 2021 at 18:29)</a>:</h4>
<p>I think the important part is guaranteeing "bytes".</p>



<a name="242965551"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/242965551" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#242965551">(Jun 17 2021 at 02:16)</a>:</h4>
<p>If the conversion function is called <code>as_bytes()</code> and returns <code>&amp;[u8]</code> then that should provide the needed access without making any promises about encoding</p>



<a name="243340286"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/243340286" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> LingM <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#243340286">(Jun 21 2021 at 05:05)</a>:</h4>
<p>Mostly for the record:</p>
<p><span class="user-mention silent" data-user-id="239881">Josh Triplett</span> <a href="#narrow/stream/219381-t-libs/topic/OsStr.20and.20WTF-8/near/242821430">said</a>:</p>
<blockquote>
<p>On Windows it's WTF-8. On UNIX platforms, it's bytes, no guarantee about encoding.<br>
You're guaranteed in both cases that you can convert UTF-8 to an OsString.</p>
</blockquote>
<p>NUL is a valid UTF-8 byte, but at least on Unix not a valid OsString byte, so that conversion can in fact fail.</p>
<p><span class="user-mention silent" data-user-id="239881">Josh Triplett</span> <a href="#narrow/stream/219381-t-libs/topic/OsStr.20and.20WTF-8/near/242822291">said</a>:</p>
<blockquote>
<p>We should allow an infallible conversion from OsStr to [u8], and a fallible conversion from [u8] to OsStr.<br>
(That conversion can only fail on Windows.)</p>
</blockquote>
<p>Similar to the above, [u8] could contain NUL, so the [u8] -&gt; OsStr conversion can fail on Unix systems, too.</p>



<a name="243344630"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/243344630" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Chris Denton <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#243344630">(Jun 21 2021 at 06:42)</a>:</h4>
<p>Surprisingly, OsStr can contain null</p>



<a name="243391881"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/243391881" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> LingM <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#243391881">(Jun 21 2021 at 14:38)</a>:</h4>
<p>Well, guess I'm one of today's luck 10k then...</p>



<a name="243412751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/243412751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#243412751">(Jun 21 2021 at 17:08)</a>:</h4>
<p>Looks like std checks for inner nul when it adds the nul terminator to an <code>OsStr</code> <a href="https://github.com/rust-lang/rust/blob/6a5b97adb4da4b0f913d19765e91d4322ef6542e/library/std/src/sys/unix/fs.rs#L911-L913">https://github.com/rust-lang/rust/blob/6a5b97adb4da4b0f913d19765e91d4322ef6542e/library/std/src/sys/unix/fs.rs#L911-L913</a></p>



<a name="243414365"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/219381-t-libs/topic/OsStr%20and%20WTF-8/near/243414365" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Chris Denton <a href="https://rust-lang.github.io/zulip_archive/stream/219381-t-libs/topic/OsStr.20and.20WTF-8.html#243414365">(Jun 21 2021 at 17:21)</a>:</h4>
<p>Yep, the std does a null check before using it in an OS API. But <code>OsStr</code> itself is allowed to contain nulls. E.g. <code>OsStr::new("Hello\0World!")</code>.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>